Papers with error analysis
NLP Lean Programming Framework: Developing NLP Applications More Effectively (N18-5)
Copied to clipboard
| Challenge: | NLPf is a framework for creating custom natural language processing models and pipelines by utilizing common software development build systems. |
| Approach: | They propose a framework for creating custom NLP models and pipelines by utilizing common software development build systems. |
| Outcome: | This framework allows developers to train and integrate domain-specific NLP pipelines into their applications seamlessly. |
Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge (2021.starsem-1)
Copied to clipboard
Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, Philippe Blache
| Challenge: | Prior work has explored the ability of computational models to predict word semantic fit with a given predicate. |
| Approach: | They compare Transformers Language Models to SDM to assess their performance . they found that TLMs do not capture important aspects of event knowledge . people can discriminate between typical and atypical events, they say . |
| Outcome: | The proposed models can achieve comparable performance to SDM, but they lack important aspects of event knowledge. |
IMSurReal: IMS at the Surface Realization Shared Task 2019 (D19-63)
Copied to clipboard
| Challenge: | a system for shallow and deep completion is presented for the Surface Realization Shared Task 2019 . the system achieves state-of-the-art performance without using external data. |
| Approach: | They propose a surface realization system that takes five steps without external data . they perform detailed error analysis revealing correlation between word order freedom and difficulty . |
| Outcome: | The proposed system achieves state-of-the-art without external data . it achieves highest BLEU scores on tokenized text and human evaluation on four languages . |
Token Sequence Labeling vs. Clause Classification for English Emotion Stimulus Detection (2020.starsem-1)
Copied to clipboard
| Challenge: | Emotion stimulus detection is the task of finding the cause of an emotion in a textual description. |
| Approach: | They propose to evaluate whether clause classification or token sequence labeling is better for emotion stimulus detection in English. |
| Outcome: | The proposed framework compares clause classification and token sequence labeling on four English datasets. |
RankME: Reliable Human Ratings for Natural Language Generation (N18-2)
Copied to clipboard
| Challenge: | Existing studies have shown that human evaluation for natural language generation often suffers from inconsistent user ratings. |
| Approach: | They propose a rank-based magnitude estimation method which combines continuous scales and relative assessments to improve the reliability of human ratings. |
| Outcome: | The proposed method significantly improves the reliability and consistency of human ratings compared to traditional evaluation methods. |
The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models (2020.emnlp-demos)
Copied to clipboard
Ian Tenney, James Wexler, Jasmijn Bastings, Tolga Bolukbasi, Andy Coenen, Sebastian Gehrmann, Ellen Jiang, Mahima Pushkarna, Carey Radebaugh, Emily Reif, Ann Yuan
| Challenge: | Existing tools for modeling and understanding models are limited . existing tools can assist practitioners in understanding and evaluating models . |
| Approach: | They present an open-source platform for visualization and understanding of NLP models. |
| Outcome: | The language interpretability tool (lit) is an open-source platform for visualization and understanding of NLP models. |
PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution (D18-1)
Copied to clipboard
| Challenge: | Existing methods for coreference resolution are based on word2vec-like representations of entities. |
| Approach: | They propose a large-scale English dataset for coreference resolution . they use 38K documents and 12.5M words from English-speaking preschoolers . |
| Outcome: | The proposed dataset is more efficient with higher training-test overlap than OntoNotes . the study also shows that mention detection and clustering are more efficient on PreCo . |
AGReE: A system for generating Automated Grammar Reading Exercises (2022.emnlp-demos)
Copied to clipboard
| Challenge: | AGReE is a system that generates multiple-choice grammar practice items . common core standards for K-12 English literacy include grammar as a learning outcome . |
| Approach: | They propose a system that generates multiple-choice grammar practice exercises that can be completed while reading. |
| Outcome: | The proposed grammar-reading exercise system can be completed while reading . it offers immediate feedback, similar to a more formal incentive system . |
Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking (2020.starsem-1)
Copied to clipboard
| Challenge: | Existing methods for dialog state tracking are ontology-based and ontologie-free . however, it is not clear enough which slots are better handled by either of the two methods . |
| Approach: | They propose a dual-strategy model that integrates both ontology-based and ontological-free methods. |
| Outcome: | The proposed model outperforms the existing model on noisy and cleaner datasets. |
Speak up, Fight Back! Detection of Social Media Disclosures of Sexual Harassment (N19-3)
Copied to clipboard
| Challenge: | #MeToo movement provides platform to narrate personal experiences of sexual harassment. |
| Approach: | They propose a three-part ULMFiT architecture to tackle text subtleties in a classification task . they propose to annotate a manually annotated real-world dataset to test their approach . |
| Outcome: | The proposed model outperforms existing models that rely on handcrafted stylistic features and is more accurate than generic models. |
“Hold on honey, men at work”: A semi-supervised approach to detecting sexism in sitcoms (2021.acl-srw)
Copied to clipboard
| Challenge: | sexist dialogue in sitcoms is an important part of society's development, according to Sink and Mastro (2017). |
| Approach: | They propose a semi-supervised text classification model that automatically detects instances of sexism in popular sitcom dialogues. |
| Outcome: | The proposed model outperforms deep learning-based systems in detecting sexist dialogues over time and shows that sexism decreases over the years. |
SNAP-BATNET: Cascading Author Profiling and Social Network Graphs for Suicide Ideation Detection on Social Media (N19-3)
Copied to clipboard
Rohan Mishra, Pradyumn Prakhar Sinha, Ramit Sawhney, Debanjan Mahata, Puneet Mathur, Rajiv Ratn Shah
| Challenge: | Suicide is a leading cause of death among youth worldwide and currently only uses text-based cues to detect suicidal ideation. |
| Approach: | They propose a deep learning based model to extract text-based features from tweets and a novel Feature Stacking approach to combine other community-based information. |
| Outcome: | The proposed model outperforms existing models on an annotated dataset of tweets using a three-phase strategy and proposes a novel Feature Stacking approach to combine other community-based information such as historical author profiling and graph embeddings. |
Evaluating Structured Output Robustness of Small Language Models for Open Attribute-Value Extraction from Clinical Notes (2025.acl-srw)
Copied to clipboard
| Challenge: | Comparative analysis of structured outputs generated by small language models for open attribute-value extraction from clinical notes . structure of outputs improves with targeted prompting and larger models, but declines for longer documents and certain note types. |
| Approach: | They compare the parsability of structured outputs generated by small language models for open attribute-value extraction from clinical notes. |
| Outcome: | The proposed model performs well in open attribute-value extraction tasks, but fails to parse for longer documents and note types. |
DRS Parsing as Sequence Labeling (2022.starsem-1)
Copied to clipboard
| Challenge: | a new semantic parser for English, German, Italian, and Dutch discourse representation structures is developed . we present a system that maps tokens to finite set of meaning fragments and is more transparent . a comprehensive error analysis highlights areas for future work on semantic parses . |
| Approach: | They propose a fully trainable semantic parser for English, German, Italian, and Dutch discourse representation structures that maps each token to one of a finite set of meaning fragments. |
| Outcome: | The proposed system is more transparent and useful for human-in-the-loop annotations. |
ConvLab-2: An Open-Source Toolkit for Building, Evaluating, and Diagnosing Dialogue Systems (2020.acl-demos)
Copied to clipboard
Qi Zhu, Zheng Zhang, Yan Fang, Xiang Li, Ryuichi Takanobu, Jinchao Li, Baolin Peng, Jianfeng Gao, Xiaoyan Zhu, Minlie Huang
| Challenge: | ConvLab-2 inherits Convlab's framework but integrates more powerful dialogue models and supports more datasets. |
| Approach: | They present ConvLab-2, an open-source toolkit that enables researchers to build task-oriented dialogue systems with state-of-the-art models and perform an end-to-end evaluation. |
| Outcome: | The new tool inherits ConvLab's framework and extends it by integrating many recently proposed state-of-the-art dialogue models. |
Extractive NarrativeQA with Heuristic Pre-Training (D19-58)
Copied to clipboard
| Challenge: | Automated question answering (QA) from text remains a challenge for humans . a striking gap exists between machine and human performance on NLP tasks . |
| Approach: | They propose a heuristic extractive version of a data set to solve the problem of answer extraction rather than generation. |
| Outcome: | The proposed model outperforms previous models on summary-level QA from full narratives and on the METEOR metric. |
What about “em”? How Commercial Machine Translation Fails to Handle (Neo-)Pronouns (2023.acl-long)
Copied to clipboard
| Challenge: | Wrong pronoun translations can discriminate against marginalized groups, e.g., non-binary individuals. |
| Approach: | They compare 3rd-person pronoun translations to five other languages . they propose to address gender exclusivity in future research . |
| Outcome: | The proposed method compares translations of gendered vs. gender-neutral pronouns from english to five other languages and vice versa. |
OLEA: Tool and Infrastructure for Offensive Language Error Analysis in English (2023.eacl-demo)
Copied to clipboard
| Challenge: | State-of-the-art models for identifying offensive language fail to generalize over nuanced or implicit cases of offensive and hateful language. |
| Approach: | They propose an open-source Python library for error analysis in the context of offensive language detection. |
| Outcome: | OLEA provides tools for error analysis in the context of detecting offensive language in English. |
An Error Analysis Framework for Shallow Surface Realization (2021.tacl-1)
Copied to clipboard
| Challenge: | BLEU and METEOR metrics fail to provide information on which linguistic factors impact performance of natural language generation models. |
| Approach: | They propose a framework for error analysis which permits identifying which features of the input affect the models’ results. |
| Outcome: | The proposed framework improves the performance of 174 system runs submitted to the Multilingual SR shared tasks. |
Rethinking Tabular Data Understanding with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are capable of various tasks, yet their capability in interpreting and reasoning over tabular data remains an underexplored area. |
| Approach: | They propose a method for table structure normalization to improve model performance . they propose aggregation of multiple reasoning pathways to improve performance based on textual and symbolic reasoning. |
| Outcome: | The proposed method improves performance on symbolic reasoning tasks with textual reasoning slightly outperforming symbolic reasoning on tables. |
Probing for Hyperbole in Pre-Trained Language Models (2023.acl-srw)
Copied to clipboard
| Challenge: | Hyperbole is a common figure of speech that involves the use of exaggerated language for emphasis or effect. |
| Approach: | They conduct edge and minimal description length probing experiments on three pre-trained language models to explore the extent to which hyperbolic information is encoded . they also annotate 63 hyperbole sentences from the HYPO dataset according to an operational taxonomy to conduct an error analysis to explore encoding of different hyperboli categories. |
| Outcome: | The results show that hyperbole is encoded in a limited extent in pre-trained models and mostly in the final layers. |
Azimuth: Systematic Error Analysis for Text Classification (2022.emnlp-demos)
Copied to clipboard
Gabrielle Gauthier-melancon, Orlando Marquez Ayala, Lindsay Brin, Chris Tyler, Frederic Branchaud-charron, Joseph Marinier, Karine Grande, Di Le
| Challenge: | Azimuth is an open-source tool to perform error analysis for text classification. |
| Approach: | They propose an open-source tool to perform error analysis for text classification . they propose to combine a range of ML techniques to facilitate the error analysis stage . |
| Outcome: | The proposed approach helps AI practitioners discover and address areas where the model does not generalize by leveraging and integrating a range of ML techniques. |
An efficient method for Natural Language Querying on Structured Data (2023.acl-industry)
Copied to clipboard
| Challenge: | a new approach to NLQ on structured data is based on text-to-SQL type semantic parsing . domain classification, domain classification and domain classification are the main tasks . semantic parsed queries are less common when information is in structured form . |
| Approach: | They propose an efficient and reliable approach to natural language Querying on databases . they use domain classification, domain classification and slot/entity extraction to query a DB . |
| Outcome: | The proposed approach simplifies the NLQ on structured data problem to the following "bread and butter" tasks. |
CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation (2025.findings-acl)
Copied to clipboard
Guangya Yu, Yanhao Li, Zongying Jiang, Yuxiong Jin, Li Dai, Yupian Lin, Ruihui Hou, Weiyan Zhang, Yongqi Fan, Qi Ye, Jingping Liu, Tong Ruan
| Challenge: | Medical quality control indicators are essential to assess the qualifications of healthcare institutions for medical services. |
| Approach: | They propose a Chinese electronic medical records-based dataset for MQCIC and propose CF-IR method that disentangles clinical fact verification and inferential rule reasoning actions. |
| Outcome: | The proposed method outperforms Chain-of-Thought methods on 20 representative LLMs, covering general and medical models. |
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022.emnlp-demos)
Copied to clipboard
| Challenge: | Existing tools to evaluate long text outputs are lacking in the field of NLP . human rating and error analysis remains a crucial component for any evaluation of long text generation. |
| Approach: | They propose a web-based toolkit to collect fine-grained error annotations for long texts . they use a taxonomy to identify errors and assign them to text spans . |
| Outcome: | The proposed tool can be used to evaluate the coherence of long generated summaries. |
A Checkpoint on Multilingual Misogyny Identification (2022.acl-srw)
Copied to clipboard
| Challenge: | a study on hate speech against minorities in Italian tweets found that 1 women are the most targeted group. |
| Approach: | They propose to train monolingual transformers and multilingual transformer models with monolingual data in English, Italian, and Spanish to detect misogyny in tweets. |
| Outcome: | The proposed model achieves state-of-the-art on English, Italian, and Spanish. |
Inducing Stereotypical Character Roles from Plot Structure (2021.emnlp-main)
Copied to clipboard
| Challenge: | Stereotypical character roles are important aids to narrative understanding and are often referred to as archetypes or dramatis personae. |
| Approach: | They propose an unsupervised method for learning stereotypical roles given only structural plot information using Vladimir Propp’s structural theory of Russian folktales. |
| Outcome: | The proposed method induces six out of seven of Vladimir Propp’s dramatis personae with F1 measures of up to 0.70 (0.58 average), with an additional category for minor characters. |
Transition-Based Chinese AMR Parsing (N18-2)
Copied to clipboard
| Challenge: | Abstract Meaning Representation (AMR) is a semantic representation where the meaning of a sentence is encoded as a rooted, directed and acyclic graph. |
| Approach: | They propose a transition-based AMR parsing framework for Chinese to be used in the next generation of AMR. |
| Outcome: | The proposed parser is based on the Chinese AMR bank. |
Framing Named Entity Linking Error Types (L18-1)
Copied to clipboard
| Challenge: | Named Entity Linking (NEL) and relation extraction forms the backbone of Knowledge Base Population tasks. |
| Approach: | They propose a taxonomy to frame common errors and apply it to four well-known Named Entity Linking systems. |
| Outcome: | The proposed taxonomy was applied to four well-known Named Entity Linking systems on three gold standards. |
ViHOS: Hate Speech Spans Detection for Vietnamese (2023.eacl-main)
Copied to clipboard
| Challenge: | Increasing use of social networking sites can cause problems for human moderators to review tagged comments. |
| Approach: | They present a dataset that contains 26k spans on 11k comments and detailed annotation guidelines . they also provide definitions of hateful and offensive spans in Vietnamese comments . |
| Outcome: | The proposed dataset shows that it is difficult to detect specific types of spans in the dataset . the dataset is the first human-annotated corpus containing 26k spans on 11k comments . |
Modeling Semantic Plausibility by Injecting World Knowledge (N18-2)
Copied to clipboard
| Challenge: | Existing models for semantic plausibility are based on distributional data, but injecting knowledge about entity properties provides a substantial performance boost. |
| Approach: | They propose to inject manually elicited knowledge about entity properties into a dataset to improve plausibility models. |
| Outcome: | The proposed dataset is a great testbed for semantic plausibility models . it shows that injection of knowledge about entity properties improves performance . |
GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generation (2025.findings-emnlp)
Copied to clipboard
Wen Ye, Zhaocheng Liu, Gui Yuwei, Tingyu Yuan, Yunyue Su, Bowen Fang, Chaoyang Zhao, Qiang Liu, Liang Wang
| Challenge: | Existing methods for text-to-image synthesis lack systematic error analysis and refinement strategies, resulting in limited reliability and effectiveness. |
| Approach: | They propose a plug-and-play multi-agent system called GenPilot that integrates error analysis, clustering-based adaptive exploration, fine-grained verification and a memory module for iterative optimization. |
| Outcome: | The proposed method improves text consistency and structural coherence on images with a plug-and-play system. |
How About Kind of Generating Hedges using End-to-End Neural Models? (2023.acl-long)
Copied to clipboard
| Challenge: | hedging is a strategy for softening the impact of a statement in conversation. |
| Approach: | They propose to fine-tune state-of-the-art language models trained on human-human tutoring data and then use a hedge classifier to select the candidate that best matches the expected hedging strategy. |
| Outcome: | The proposed model is feasible in a noisy environment with reranking, and it is compared with other approaches. |
Systems’ Agreements and Disagreements in Temporal Processing: An Extensive Error Analysis of the TempEval-3 Task (L18-1)
Copied to clipboard
| Challenge: | Temporal Processing systems are crucial for timelines and storylines . TempEval-3 is the latest evaluation campaign on open-domain TP in english . |
| Approach: | They present a Temporal Processing system that incorporates high level lexical semantic features and uses them to evaluate temporal relation classification. |
| Outcome: | The proposed system achieves the best scores for event detection and temporal relation classification from raw text, but the errors are not as robust as previous systems. |
LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing Text-to-SQL models fail to address schema linking problems in large-scale multi-database environments. |
| Approach: | They propose a framework that aims to enable non-expert users to retrieve data effortlessly . they highlight four core errors leading to schema linking failures . |
| Outcome: | The proposed framework outperforms baselines on all schema linking metrics. |
Negation Detection in Dutch Spoken Human-Computer Conversations (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing negation detection methods in English are not available. |
| Approach: | They propose to annotate a Dutch dialogue corpus with negation cues and their scopes. |
| Outcome: | The proposed method can detect negation cues and scope in Dutch dialogues with high precision and recall. |
LM-CORE: Language Models with Contextually Relevant External Knowledge (2022.findings-naacl)
Copied to clipboard
| Challenge: | Large pre-trained language models can capture factual knowledge in their parameters but storing large amounts of knowledge in the model parameters is sub-optimal given the ever-growing amounts of information and resource requirements. |
| Approach: | They propose a framework that provides explicit access to contextually relevant structured knowledge to the model and train it to use that knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art knowledge-enhanced language models on knowledge probing tasks and can handle knowledge updates. |
Liputan6: A Large-scale Indonesian Dataset for Text Summarization (2020.aacl-main)
Copied to clipboard
| Challenge: | Despite having the fourth largest speaker population in the world, 1 Indonesian is under-represented in NLP. |
| Approach: | They propose to use a large-scale Indonesian summarization dataset to test extractive and abstractive summarizing methods. |
| Outcome: | The proposed methods are compared with multilingual and monolingual BERT-based models. |
Joint End-to-end Semantic Proto-role Labeling (2023.acl-short)
Copied to clipboard
| Challenge: | Existing systems for semantic proto-role labeling assign binary properties to arguments based on agent-like or patient-like properties. |
| Approach: | They propose to use a deep transformer model to model the performance of semantic proto-role labeling . they propose to include an error analysis to understand correlations between system stages . |
| Outcome: | The proposed system is robust in the presence of predicted arguments, the authors show . the proposed system also reduces annotation errors, the researchers conclude . |
Contextual Semantic Parsing for Multilingual Task-Oriented Dialogues (2023.eacl-main)
Copied to clipboard
| Challenge: | Existing methods for predicting state of a conversation are limited to a few languages . a method that can be applied to other languages will benefit the large population of speakers of many other languages. |
| Approach: | They propose to automatically translate large-scale dialogue data sets in one language to produce an effective semantic parser for other languages using machine translation. |
| Outcome: | The proposed model reduces the compounding effect of translation errors without harming the accuracy in practice. |
Re-Examining FactBank: Predicting the Author’s Presentation of Factuality (2022.coling-1)
Copied to clipboard
| Challenge: | Previously published results on FactBank are no longer valid. |
| Approach: | They propose to correct a subset of FactBank data to improve performance . they use multiple training paradigms, data smoothing techniques, and polarity classifiers . |
| Outcome: | The proposed model improves performance on the FactBank dataset. |
Errudite: Scalable, Reproducible, and Testable Error Analysis (P19-1)
Copied to clipboard
| Challenge: | Errudite is an interactive tool for informative error analysis. |
| Approach: | They propose an interactive tool for model and task agnostic error analysis . they propose a model-agnosed approach to model and tasks error analysis with Errudite . |
| Outcome: | The proposed tool enables reproducible error analyses with less effort and ambiguities in previous tools. |
Contextualized Query Embeddings for Conversational Search (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to conversational search use multiple inference pipelines that require long inference times . despite their effectiveness, such a pipeline often includes multiple neural models that require longer inference time. |
| Approach: | They propose to integrate conversational query reformulation directly into a dense retrieval model . they use a dataset with pseudo-relevance labels to overcome the lack of training data . |
| Outcome: | The proposed model rewrites conversational queries as dense representations in conversational search and open-domain question answering datasets. |
Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging? (2022.naacl-main)
Copied to clipboard
| Challenge: | Recent Part-Of-Speech (POS) induction models assume certain independence assumptions that do not hold in real languages. |
| Approach: | They propose a Masked Part-of-Speech Model (MPoSM) that can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
| Outcome: | The proposed model can model arbitrary tag dependency and perform POS induction through the objective of masked POS reconstruction. |
Monte Carlo Syntax Marginals for Exploring and Using Dependency Parses (N18-1)
Copied to clipboard
| Challenge: | Dependency parsing research focuses on improving accuracy of single-tree predictions . ambiguity is inherent to natural language syntax, and communicating it is important for error analysis . |
| Approach: | They propose a transition sampling algorithm to sample from the full joint distribution of parse trees defined by a model and demonstrate its usefulness. |
| Outcome: | The proposed method can be used to propagate parse uncertainty to two downstream applications. |
PoMo: Generating Entity-Specific Post-Modifiers in Context (N19-1)
Copied to clipboard
Jun Seok Kang, Robert Logan, Zewei Chu, Yang Chen, Dheeru Dua, Kevin Gimpel, Sameer Singh, Niranjan Balasubramanian
| Challenge: | Using crowdsourcing, we show that contextual relevance is necessary for accurate post-modifier generation. |
| Approach: | They introduce entity post-modifier generation as an instance of a collaborative writing task . they build a post- modifier dataset from news articles that provides contextually relevant information about the target entity. |
| Outcome: | The proposed system can generate a post-modifier phrase that provides contextually relevant information about the target entity. |
Toxic Language Detection in Social Media for Brazilian Portuguese: New Dataset and Multilingual Analysis (2020.aacl-main)
Copied to clipboard
| Challenge: | Hate speech and toxic comments are a common concern of social media platform users . identifying toxic comments is important for studying and preventing the proliferation of toxicity in social media. |
| Approach: | They propose to use Brazilian Portuguese to analyze toxic or non-toxic tweets . they propose to analyze tweets as toxic or in different types of toxicity . |
| Outcome: | The proposed model achieves 76% macro-F1 score using monolingual data in the binary case. |
Unveiling Dual Quality in Product Reviews: An NLP-Based Approach (2025.acl-industry)
Copied to clipboard
| Challenge: | Dual quality is a problem where products with identical ingredients or characteristics are sold under the same brand and similar packaging in different markets, but are significantly altered in composition or quality parameters. |
| Approach: | They propose to use natural language processing to detect inconsistent product quality by analyzing a Polish-language dataset and using different approaches. |
| Outcome: | The proposed approach can detect and address inconsistent product quality in Polish and other languages. |
A Speech Recognizer for Frisian/Dutch Council Meetings (2022.lrec-1)
Copied to clipboard
Martijn Bentum, Louis ten Bosch, Henk van den Heuvel, Simone Wills, Domenique van der Niet, Jelske Dijkstra, Hans Van de Velde
| Challenge: | During council meetings both Frisian and Dutch are spoken, and code switching between both languages shows up frequently. |
| Approach: | They develop a bilingual Frisian/Dutch speech recognizer for council meetings in Fryslân (the Netherlands) based on an existing Frisian and Dutch speech recognized by FAME!, which was trained and tested on radio broadcasts. |
| Outcome: | The new recognizer is based on an existing speech recognizer for Frisian and Dutch named FAME!, which was trained and tested on radio broadcasts. |
ZSEE: A Dataset based on Zeolite Synthesis Event Extraction for Automated Synthesis Platform (2024.findings-naacl)
Copied to clipboard
| Challenge: | Automated synthesis of zeolite holds great significance for attaining economic and environmental benefits. |
| Approach: | They propose an event extraction task to mine structural synthesis actions from experimental narratives for modular automated synthesis. |
| Outcome: | The proposed method can significantly expedite automated synthesis of zeolites owing to its machine readability. |
Exploiting Pre-Ordering for Neural Machine Translation (L18-1)
Copied to clipboard
| Challenge: | Existing studies have shown that Neural Machine Translation suffers from the problems that some source words are mistakenly translated for multiple times . |
| Approach: | They propose a pre-ordering approach to solve the under-translation problem by pre-ordnanced source sentences and position embedding to enhance monotone translation. |
| Outcome: | The proposed method significantly improves translation quality by 2.43 BLEU points on Chinese-to-English translation. |
Learning From Revisions: Quality Assessment of Claims in Argumentation at Scale (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing research on predicting argument quality based on subjective assessments of human annotators ignores this limitation. |
| Approach: | They propose to compare different revisions of the same claim to assess their quality . they use logistic regression and transformer-based neural networks to learn quality indicators . |
| Outcome: | The proposed tasks show that the learned indicators generalize well across topics. |
OptiVerse: A Comprehensive Benchmark towards Optimization Problem Solving (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing benchmarks focus on Mathematical Programming and Combinatorial Optimization, hindering comprehensive evaluation. |
| Approach: | They propose a benchmarking tool that compares 1,000 curated optimization problems across three difficulty levels. |
| Outcome: | The proposed model improves performance on hard problems while maintaining 27% accuracy. |
Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans (2022.lrec-1)
Copied to clipboard
| Challenge: | Annotations with incorrect label or boundaries count as two errors instead of one, despite being closer to the target annotation than false positives or false negatives. |
| Approach: | They propose an algorithm for error identification in flat and multi-level annotations and propose a procedure for calculating meaningful precision, recall, and F1-scores based on the more fine-grained error types. |
| Outcome: | The proposed procedure prevents double penalties and allows for a more detailed error analysis, providing more insight into the actual weaknesses of a system. |
An Algerian Corpus and an Annotation Platform for Opinion and Emotion Analysis (2020.lrec-1)
Copied to clipboard
| Challenge: | Currently, there are more than 4 billion Internet users worldwide . the number of social media users in Algeria has tripled over a year . |
| Approach: | They propose a platform for crowdsourcing annotation of tweets at different levels of granularity. |
| Outcome: | The proposed platform can be used to create the largest Algerian dialect subjectivity lexicon of about 9,000 entries. |
BREAK: Breaking the Dialogue State Tracking Barrier with Beam Search and Re-ranking (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods for dialogue state tracking still have a JGA of 60% on MultiWOZ 2.1 . break framework provides a simple yet effective way to generate dialogue state candidates . |
| Approach: | They propose a framework that generates k-best dialogue state candidates with beam search and re-ranks them to select the correct dialogue state. |
| Outcome: | The proposed framework pushes the joint goal accuracy to 80-90% on MultiWOZ 2.1-2.4. |
An Empirical Exploration of Local Ordering Pre-training for Structured Prediction (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that pre-training contextualized encoders with language model objectives is effective for structured prediction. |
| Approach: | They propose a semi-supervised method for pre-training contextualized encoders with language model objectives. |
| Outcome: | The proposed method is effective on three typical structured prediction tasks in four languages. |
Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts (2024.findings-naacl)
Copied to clipboard
Chenghao Yang, Tuhin Chakrabarty, Karli Hochstatter, Melissa Slavin, Nabila El-Bassel, Smaranda Muresan
| Challenge: | Experimental results show that identifying the phases of opioid use disorder is highly contextual and challenging. |
| Approach: | They analyze 2500 opioid-related posts from various subreddits labeled with six different phases of opioid use . they annotate span-level extractive explanations and critically evaluate state-of-the-art models in a supervised, few-shot, or zero-shot setting. |
| Outcome: | The proposed models improve classification accuracy and quality of the extracted explanations. |
Finding Diamonds in Conversation Haystacks: A Benchmark for Conversational Data Retrieval (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Our work identifies unique challenges in conversational data retrieval . large language model-based systems operate through open-ended interactions without predefined specifications. |
| Approach: | They propose a benchmark to evaluate systems that retrieve conversation data for product insights. |
| Outcome: | The benchmark provides a reliable standard for measuring conversational data retrieval performance. |
Optimizing Deeper Transformers on Small Datasets (2021.acl-long)
Copied to clipboard
Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, Yanshuai Cao
| Challenge: | a common belief that training deep transformers from scratch requires large datasets is wrong . however, with proper initialization and optimization, the benefits of very deep transformer can carry over to challenging tasks with small datasets. |
| Approach: | They train 48 layers of transformers from pre-trained RoBERTa and 24 relation-aware layers from scratch. |
| Outcome: | The proposed scheme achieves state-of-the-art performance on a text-to-sql parsing benchmark . it uses 24 fine-tuned layers from pre-trained RoBERTa and 24 relation-aware layers from scratch . |
GLEN: General-Purpose Event Detection for Thousands of Types (2023.emnlp-main)
Copied to clipboard
| Challenge: | ACE 2005 2 is the first large-scale event extraction dataset with 205K event mentions and 3,465 different types. |
| Approach: | They propose to use the DWD Overlay to map PropBank rolesets to a large distantlysupervised training dataset with partial labels to make event extraction more accessible. |
| Outcome: | The proposed model performs better than baselines including InstructGPT and ACE 2005 2 despite being 18 years old . key limitations of ACE include its small event ontology of 33 types, small dataset size of around 600 documents and restricted domain (with a significant portion concentrated on military conflicts). |
Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement (2025.emnlp-industry)
Copied to clipboard
Haotan Guo, Jianfei He, Jiayuan Ma, Hongbin Na, Zimu Wang, Haiyang Zhang, Qi Chen, Wei Wang, Zijing Shi, Tao Shen, Ling Chen
| Challenge: | Phonetic Cloaking Replacement (PCR) is a problem in content moderation in China. |
| Approach: | They organize PCR into a four-way surface-form taxonomy and compile PCR-ToxiCN, a dataset of 500 phonetically cloaked offensive posts gathered from the RedNote platform. |
| Outcome: | The proposed model achieves only an F1-score and zero-shot chain-of-thought prompting pushes performance even lower. |
ExtEnD: Extractive Entity Disambiguation (2022.acl-long)
Copied to clipboard
| Challenge: | Entity disambiguation (ED) is a task in natural language processing that requires a large pre-trained language model to perform. |
| Approach: | They propose a local formulation for Entity Disambiguation (ED) that frames this task as a text extraction problem and propose two Transformer-based architectures that implement it. |
| Outcome: | The proposed model outperforms all its competitors in terms of data efficiency and raw performance on 4 out of 4 benchmarks. |
It’s All About the Confidence: An Unsupervised Approach for Multilingual Historical Entity Linking using Large Language Models (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to EL for historical texts require substantial training data or rely on domain-specific rules that limit scalability. |
| Approach: | They propose an unsupervised ensemble approach combining a Small Language Model and an LLM for historical EL. |
| Outcome: | The proposed approach outperforms state-of-the-art models on four established benchmarks in six European languages from the 19th and 20th centuries. |
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs (2023.emnlp-main)
Copied to clipboard
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, Yongbin Li
| Challenge: | Recent research has demonstrated that Large Language Models (LLMs) can enhance their capabilities by utilizing external tools. |
| Approach: | They propose a runnable evaluation system consisting of 73 API tools and an annotation system for 314 tool-use dialogues with 753 API calls. |
| Outcome: | The proposed benchmark assesses the effectiveness of existing LLMs by analyzing 314 tool-use dialogues with 753 API calls. |
Scoring and Classifying Implicit Positive Interpretations: A Challenge of Class Imbalance (C18-1)
Copied to clipboard
| Challenge: | a reimplementation of a system on detecting implicit positive meaning from negated statements is reported . a baseline taking the mean score or most frequent class is hard to beat because of class imbalance in the dataset. |
| Approach: | They propose a system to detect implicit positive meaning from negated statements . they convert the scores into classes and report their results on regression and classification tasks . |
| Outcome: | The proposed system is hard to beat because of class imbalance in the dataset. |
On the Summarization of Consumer Health Questions (P19-1)
Copied to clipboard
| Challenge: | Question understanding is one of the main challenges in question answering. |
| Approach: | They propose to use semantic augmentation to augment question datasets to improve their performance. |
| Outcome: | The proposed model outperforms sequence-to-sequence attentional models on the medical question summarization task with a ROUGE-1 score of 44.16%. |
ETPC - A Paraphrase Identification Corpus Annotated with Extended Paraphrase Typology and Negation (L18-1)
Copied to clipboard
| Challenge: | Extended Paraphrase Typology addresses limitations of existing typologies . extended typology provides better means for evaluation and error analysis . |
| Approach: | a new typology copes with non-paraphrase pairs in paraphrase identification corpora, a paper proposes . a large corpus annotated with atomic paraphrase types is the largest to date . |
| Outcome: | The Extended Paraphrase Typology (EPT) and the Extended Typology Paraphrase Corpus (ETPC) address practical limitations of existing paraphrase typologies. |
From Toxicity in Online Comments to Incivility in American News: Proceed with Caution (2021.eacl-main)
Copied to clipboard
| Challenge: | Existing tools for quantifying incivility online, in news and in congressional debates are inadequate for the analysis of incivility in news. |
| Approach: | They develop a Jigsaw Perspective API to quantify incivility in news . they show that toxicity models are inadequate for the analysis of incivility in news. |
| Outcome: | The Jigsaw Perspective API detects incivility on a corpus of American news articles. |
Are People Located in the Places They Mention in Their Tweets? A Multimodal Approach (2022.coling-1)
Copied to clipboard
| Challenge: | Experimental results show that a neural architecture that combines both modalities yields better results. |
| Approach: | They propose a neural architecture that combines both modalities to solve the problem of determining whether people are located in tweets. |
| Outcome: | The proposed model combines both modalities to produce better results . |
Homophonic Pun Generation with Lexically Constrained Rewriting (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new method for generating puns using two homophones is needed to generate creative puns . early models for pun generation rely on templates and lack novelty. |
| Approach: | They propose a neural approach to generate homophonic puns with two meanings . they use constraint words to find the semantic incongruity and explicit negative constraints . |
| Outcome: | The proposed model achieves state-of-the-art in automatic and human evaluations. |
CLIX: Cross-Lingual Explanations of Idiomatic Expressions (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing definition generation systems are difficult to use in second language learning due to the presence of unfamiliar words and grammar. |
| Approach: | They propose to use cross-lingual explanations of idiomatic expressions to support vocabulary expansion for language learners. |
| Outcome: | The proposed system is able to explain idiomatic expressions in non-standard language. |
On the Limit of Language Models as Planning Formalizers (2025.acl-long)
Copied to clipboard
| Challenge: | Large Language Models can create plans that are neither executable nor verifiable in grounded environments. |
| Approach: | They use Large Language Models to generate a formal representation of the planning domain in some language, such as Planning Domain Definition Language (PDDL). |
| Outcome: | The proposed model outperforms the models directly generating plans while being robust to lexical perturbation. |
Vicinal Risk Minimization for Few-Shot Cross-lingual Transfer in Abusive Language Detection (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for few-shot cross-lingual transfer learning are limited in target languages due to the scarcity of resources. |
| Approach: | They propose a method which interpolates pairs of instances based on the angle of their representations and propose augmentation methods to enhance few-shot cross-lingual abusive language detection. |
| Outcome: | The proposed method improves few-shot cross-lingual abusive language detection in seven languages typologically distinct from English and three different domains. |
DialFact: A Benchmark for Fact-Checking in Dialogue (2022.acl-long)
Copied to clipboard
| Challenge: | Existing fact-checking models trained on non-dialogue data fail to perform well on this task. |
| Approach: | They propose a task of fact-checking in dialogue to improve fact- checking performance . they propose to use an annotated conversational claim and Wikipedia snippets as evidence . |
| Outcome: | The proposed task improves fact-checking performance in dialogue. |
Automatic Error Analysis for Document-level Information Extraction (2022.acl-long)
Copied to clipboard
| Challenge: | Document-level information extraction (IE) tasks have been revisited in earnest . evaluation of the approaches has been limited in a number of dimensions . |
| Approach: | They propose a transformation-based framework for automating error analysis in document-level event and (N-ary) relation extraction. |
| Outcome: | The proposed framework compares two state-of-the-art document-level template-filling approaches on datasets from three domains and four systems from the MUC-4 evaluation. |
Alignment over Heterogeneous Embeddings for Question Answering (N19-1)
Copied to clipboard
| Challenge: | Existing approaches for non-factoid question answering are based on heterogeneous embeddings that model text at different levels of abstraction. |
| Approach: | They propose a fast, mostly-unsupervised approach for non-factoid question answering called Alignment over Heterogeneous Embeddings (AHE) it aligns each word in the question and candidate answer with the most similar word in retrieved supporting paragraph and a meta-classifier that learns how much to trust the predictions over each representation. |
| Outcome: | The proposed approach outperforms other supervised approaches on the AI2 Reasoning Challenge dataset and the WikiQA dataset. |
ChatEL: Entity Linking with Chatbots (2024.lrec-main)
Copied to clipboard
| Challenge: | Entity Linking (EL) is a challenging task in natural language processing . existing approaches focus on creating elaborate contextual models that are unwieldy and difficult to train . |
| Approach: | They propose a framework to prompt LLMs to return accurate results for Entity Linking . they use a three-step framework to generate a set of EL models that can be open-source . |
| Outcome: | The proposed framework improves the average F1 performance across 10 datasets by more than 2%. |
On the Human-level Performance of Visual Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | Visual7W has been widely used in assessing multiple-choice visual question-answering systems. |
| Approach: | They replicated a human experiment on Visual7W to examine the human-level performance of VQA. |
| Outcome: | The results show that the better a model performs on Visual7W, the better it aligns with human-level intelligence. |
Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to Knowledge Base Question Answering focus on semantic parsing . previous work focused on selecting the correct semantic relations and not on the structure of the semantic parses . |
| Approach: | They propose to use Gated Graph Neural Networks to encode the graph structure of the semantic parse. |
| Outcome: | The proposed approach outperforms baseline models that do not explicitly model the structure. |
Open Information Extraction on Scientific Text: An Evaluation (C18-1)
Copied to clipboard
| Challenge: | Open Information Extraction (OIE) is the unsupervised creation of structured information from text. |
| Approach: | They propose to use two state-of-the-art OIE systems to evaluate the performance of OIE on scientific texts originating from 10 different disciplines. |
| Outcome: | The proposed methods perform significantly worse on scientific text than encyclopedic text. |
Geo-Seq2seq: Twitter User Geolocation on Noisy Data through Sequence to Sequence Learning (2023.findings-acl)
Copied to clipboard
| Challenge: | a new method for Twitter user geolocation rewrites noisy, multilingual location strings into structured English location names. |
| Approach: | They propose a sequence-to-sequence (seq2sequ) model that rewrites noisy location strings into structured English location names. |
| Outcome: | The proposed model can generalize well to unseen temporal data, but performance does vary by language and country. |
MacGyver: Are Large Language Models Creative Problem Solvers? (2024.naacl-long)
Copied to clipboard
Yufei Tian, Abhilasha Ravichander, Lianhui Qin, Ronan Le Bras, Raja Marjieh, Nanyun Peng, Yejin Choi, Thomas Griffiths, Faeze Brahman
| Challenge: | a new study examines the creative problem-solving capabilities of modern LLMs . it provides insight into the constrained problem- solving capabilities of both humans and AI . |
| Approach: | They use an automatically generated dataset to compare and contrast LLMs and humans to find out their creative problem-solving abilities. |
| Outcome: | The proposed dataset compares LLMs and humans in a constrained setting . it shows that humans excel in tasks they are familiar with but struggle with domain-specific knowledge . |
ABCD: A Graph Framework to Convert Complex Sentences to a Covering Set of Simple Sentences (2021.acl-long)
Copied to clipboard
| Challenge: | Existing work relies on rule-based methods dependent on parsing to identify atomic sentences. |
| Approach: | They propose a task to decompose complex sentences into simple ones . they propose atomic clauses as atomic sentences, and a graph edit task to predict edits . |
| Outcome: | The proposed model performs better than baselines on MinWiki and DeSSE. |
Biomedical Interpretable Entity Representations (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing work on general interpretable representation learning does not transfer to biomedicine . pre-trained models induce dense entity representations but are not immediately interpretable. |
| Approach: | They propose a method that exploits BIER's final sparse and intermediate dense representations to facilitate model and entity type debugging. |
| Outcome: | The proposed model performs well on biomedical tasks including disambiguation and label classification. |
Impact Analysis of the Use of Speech and Language Models Pretrained by Self-Supersivion for Spoken Language Understanding (2022.lrec-1)
Copied to clipboard
Salima Mdhaffar, Valentin Pelloin, Antoine Caubrière, Gaëlle Laperriere, Sahar Ghannay, Bassam Jabaian, Nathalie Camelin, Yannick Estève
| Challenge: | Pretrained models have been introduced for both acoustic and language modeling. |
| Approach: | They present an error analysis of pretrained models using a french MEDIA benchmark dataset. |
| Outcome: | The proposed models have been able to improve on the french MEDIA benchmark dataset, which is one of the most challenging among all benchmarks accessible to the entire research community. |
MIsA: Multilingual “IsA” Extraction from Corpora (L18-1)
Copied to clipboard
| Challenge: | In this paper, we present a collection of hypernymy relations extracted from the Wikipedia corpus in five languages. |
| Approach: | They present a collection of hypernymy relations extracted from the Wikipedia corpus in five languages . they use existing or newly defined lexico-syntactic patterns to extract hyperniyms . |
| Outcome: | The proposed tool is based on a dictionary extracted from the full Wikipedia corpus. |
Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | Negation is a fundamental operation in natural language that reverses the meaning of an expression into its opposite. |
| Approach: | They propose a sentence-level negation understanding benchmark that measures negation performance in Korean. |
| Outcome: | The proposed benchmark improves negation understanding and broader comprehension in Korean. |
Intent Classification and Slot Filling for Privacy Policies (2021.acl-long)
Copied to clipboard
| Challenge: | Sentences written in privacy policies explain privacy practices and the constituent text spans convey further specific information. |
| Approach: | They propose an English corpus of 5,250 intent and 11,788 slot annotations . they propose two alternative neural approaches to model the corpus as a sequence-to-sequence learning task. |
| Outcome: | The proposed corpus predicts intent classification and slot filling, while the sequence tagging method outperforms slot filler by a large margin. |
Multi-Aspect Transfer Learning for Detecting Low Resource Mental Disorders on Social Media (2022.lrec-1)
Copied to clipboard
| Challenge: | Mental disorders are an important and pervasive public health issue. |
| Approach: | They propose to use linguistic features to improve mental disorder detection . they propose to apply multi-aspect transfer learning to detecting disorders from social media . |
| Outcome: | The proposed methods can be used to improve mental disorder detection in the context of data scarcity and understanding the overlapping symptoms between disorders. |
Written Justifications are Key to Aggregate Crowdsourced Forecasts (2021.findings-emnlp)
Copied to clipboard
| Challenge: | aggregating crowdsourced forecasts benefits from modeling written justifications . a majority of respondents support the idea that crowds are more reliable than experts . |
| Approach: | They propose to model written justifications for crowdsourced questions by analyzing their results in a literature review. |
| Outcome: | The results show that the written justifications are beneficial to call a question throughout its life except in the last quarter. |
On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles (2021.naacl-main)
Copied to clipboard
| Challenge: | citation worthiness is an emerging research topic in the natural language processing domain . citation recommendation systems are often approached as ranking problems . |
| Approach: | They propose a hierarchical biLSTM-based model that uses two adjacent sentences to solve a citation worthiness problem. |
| Outcome: | The proposed approach can be applied to a dataset of over two million sentences and their labels. |
Zero-Shot Cross-Lingual Transfer with Meta Learning (2020.emnlp-main)
Copied to clipboard
| Challenge: | There are more than 7,000 languages spoken in the world, over 90 of which have more than 10 million native speakers each. |
| Approach: | They propose to use meta-learning to train a model on multiple languages at the same time . they use standard supervised, zero-shot cross-lingual, and few-shot crosses-lingual settings for different natural language understanding tasks. |
| Outcome: | The proposed setup improves on the state-of-the-art for a total of 15 languages. |
Translation-based Lexicalization Generation and Lexical Gap Detection: Application to Kinship Terms (2024.acl-long)
Copied to clipboard
| Challenge: | Existing methods for identifying lexical gaps have been limited . kinship terms are well-suited for investigations into lexicons and lexicals . |
| Approach: | They propose an algorithm to automatically generate concept lexicalizations based on machine translation and hypernymy relations between concepts. |
| Outcome: | Empirical evaluations show that the proposed method is more accurate than BabelNet and ChatGPT. |
Czech Dataset for Complex Aspect-Based Sentiment Analysis Tasks (2024.lrec-main)
Copied to clipboard
| Challenge: | 3.1K reviews are manually annotated for aspect-based sentiment analysis (ABSA) ABSA is a fine-grained task that aims to identify the sentiment associated with each aspect or characteristic of a text. |
| Approach: | They propose a new Czech dataset for aspect-based sentiment analysis . the new dataset is built upon the older Czech dataset . authors provide 24M reviews without annotations suitable for unsupervised learning . |
| Outcome: | The proposed dataset is built upon the older dataset, but is specifically designed for more complex tasks. |
Sarcasm Detection is Way Too Easy! An Empirical Comparison of Human and Machine Sarcasm Detection (2022.findings-emnlp)
Copied to clipboard
| Challenge: | sarcasm detection datasets focus on intended, rather than perceived sarcasm, but there is no comparison between human and machine performance. |
| Approach: | They collect author-annotated sarcasm datasets that focus on intended, rather than perceived sarcasticism . they compare human-level benchmarks to that of state-of-the-art sarkasmatic detection systems . |
| Outcome: | The proposed datasets compare human and machine performance on sarcastic tasks in English and Arabic. |
Liar, Liar, Logical Mire: A Benchmark for Suppositional Reasoning in Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | logical puzzles that involve determining identity of characters require a variety of reasoning skills. |
| Approach: | They propose a benchmark for suppositional reasoning based on knights and knaves puzzles . they show lower-performing models exhibit a diverse range of reasoning errors . |
| Outcome: | The proposed benchmark demonstrates that models struggle with suppositional reasoning . lower performing models struggle to grasp the concept of truth and lies, the study finds . |
A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing evaluations of entity linking systems often lack detailed error analysis or a closer look at the results. |
| Approach: | They evaluate existing entity linking systems and propose two new benchmarks . they characterize their strengths and weaknesses and report on reproducibility aspects . |
| Outcome: | The evaluations of existing system have strong biases and artifacts . they characterize their strengths and weaknesses and report on reproducibility aspects . |
Utilizing Subword Entities in Character-Level Sequence-to-Sequence Lemmatization Models (2020.coling-main)
Copied to clipboard
| Challenge: | a novel character-level sequence-to-sequence lemmatization model uses generic n-gram embeddings to map word/lemma pairs . semitic languages, like Arabic and Hebrew, add other challenges to handle unseen words . |
| Approach: | They propose a character-level sequence-to-sequence lemmatization model . they use generic n-gram embeddings, concatenative (stems) and templatic (roots and patterns) morphological subwords. |
| Outcome: | The proposed model outperforms other linguistically-driven models with generic n-gram embeddings . the best system handles word/lemma pairs that are both unseen in the training data . |
Compositional Questions Do Not Necessitate Multi-hop Reasoning (P19-1)
Copied to clipboard
| Challenge: | a single-hop reasoning model can solve much more of the dataset than previously thought. |
| Approach: | They propose a single-hop BERT-based RC model that achieves 67 F1 . they propose an evaluation setting where humans are not shown all paragraphs . |
| Outcome: | The proposed model achieves 67 F1—comparable to state-of-the-art multi-hop models. |
Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving (2025.findings-naacl)
Copied to clipboard
Botao Yu, Frazier N. Baker, Ziru Chen, Garrett Herb, Boyu Gou, Daniel Adu-Ampratwum, Xia Ning, Huan Sun
| Challenge: | Existing evaluations of large language models (LLMs) with tools are limited and qualitative . existing evaluations have been limited and only focus on 14 tasks focusing on compound synthesis. |
| Approach: | They propose to develop an enhanced chemistry agent over ChemCrow to improve chemistry problem solving by integrating tools into LLMs. |
| Outcome: | The proposed agent does not consistently outperform its base LLMs without tools on specialized chemistry tasks and general chemistry questions. |
PcMSP: A Dataset for Scientific Action Graphs Extraction from Polycrystalline Materials Synthesis Procedure Text (2022.findings-emnlp)
Copied to clipboard
| Challenge: | 305 open access scientific articles are used for synthesis action graphs . lack of annotated data has hindered progress in this field . |
| Approach: | They propose to annotate Polycrystalline Materials Synthesis Procedures PcMSP from 305 open access scientific articles for the construction of synthesis action graphs. |
| Outcome: | The proposed dataset contains the synthesis sentences, entity mentions and intra-sentence relations extracted from the experimental paragraphs. |
XFinBench: Benchmarking LLMs in Complex Financial Problem Solving and Reasoning (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing large language models (LLMs) lack advanced capabilities such as temporal reasoning, future forecasting, and numerical modeling. |
| Approach: | They propose a benchmarking tool to evaluate LLMs' ability to solve complex financial problems across diverse graduate-level finance topics with multi-modal context. |
| Outcome: | The proposed model improves on the o1 model but still lags behind human experts in temporal reasoning and scenario planning capabilities. |
Debate, Reflect, and Distill: Multi-Agent Feedback with Tree-Structured Preference Optimization for Efficient Language Model Enhancement (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for enhancing small models struggle to yield substantial and lasting performance gains. |
| Approach: | They propose a Debate and Reflect framework that orchestrates multi-turn debates between smaller models and stronger teacher models. |
| Outcome: | The proposed framework outperforms existing methods by a large margin in smaller models. |
Yet Another Format of Universal Dependencies for Korean (2022.coling-1)
Copied to clipboard
Yige Chen, Eunkyul Leah Jo, Yundong Yao, KyungTae Lim, Miikka Silfverberg, Francis M. Tyers, Jungyeul Park
| Challenge: | Existing dependency parsers for Korean do not perform as well as their English counterparts due to the complexity of Korean's linguistic features. |
| Approach: | They propose a morpheme-based Korean dependency parsing format and propose to adopt it to Universal Dependencies. |
| Outcome: | The proposed format outperforms parsing results for Korean UD treebanks and detailed error analysis. |
Clickbait Spoiling via Question Answering and Passage Retrieval (2022.acl-long)
Copied to clipboard
| Challenge: | Clickbait is a term used to describe posts intended to entice readers to visit a web page . clickbait spoiling is generating a short text that satisfies the curiosity induced by a clickbaiting post . |
| Approach: | They propose to use clickbait spoiling to generate a short text that satisfies curiosity . they classify the type of spoiler needed and generate appropriate spoilers . |
| Outcome: | The proposed method outperforms all other methods in generating spoilers for both types of clickbait posts. |
Can Large Language Models Win the International Mathematical Games? (2025.emnlp-main)
Copied to clipboard
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, Gianluca Moro
| Challenge: | Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts. |
| Approach: | They propose a benchmark of 2,183 high-quality mathematical problems in an open-ended format that enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
| Outcome: | The new benchmark spans seven age groups and a skill-based taxonomy and enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
Introducing Graph Context into Language Models through Parameter-Efficient Fine-Tuning for Lexical Relation Mining (2025.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models can effectively mine lexical relations between word pairs . however, graph features and semantic knowledge of pre-tried models are lacking in the task. |
| Approach: | They propose a parameter-efficient fine-tuning method which integrates graph features and semantic representations for lexical relation classification and lexic entailment tasks. |
| Outcome: | The proposed method integrates graph features and semantic representations for lexical relation mining tasks. |
Adapting Coreference Resolution Models through Active Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Neural coreference resolution models trained on one dataset may not transfer to new, low-resource domains. |
| Approach: | They investigate how to actively label coreference by sampling a small subset of data for annotators to label. |
| Outcome: | The proposed model can be more realistic when labeling spans within the same document than when annotating spans across documents. |
Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twitter (2022.emnlp-main)
Copied to clipboard
| Challenge: | Current vogue is to employ manual fact-checkers to efficiently classify and verify such data to combat this avalanche of misinformation and fake news. |
| Approach: | They propose a large-scale Twitter corpus with token-level claim spans on more than 7.5k tweets and a model that automatically detects and extracts the snippets of misinformation. |
| Outcome: | The proposed model outperforms baseline systems on several evaluation metrics, improving by 1.5 points. |
State-of-the-art Chinese Word Segmentation with Bi-LSTMs (D18-1)
Copied to clipboard
| Challenge: | A wide variety of neural-network architectures have been proposed for the task of Chinese word segmentation. |
| Approach: | They propose a bidirectional LSTM model with standard deep learning techniques and best practices for the task of Chinese word segmentation. |
| Outcome: | The proposed model outperforms models based on standard deep learning techniques and best practices on Chinese word segmentation datasets. |
Building The First English-Brazilian Portuguese Corpus for Automatic Post-Editing (2020.coling-main)
Copied to clipboard
| Challenge: | Existing corpus for automatic post-editing of English and Brazilian Portuguese is limited. |
| Approach: | They introduce a corpus for Automatic Post-Editing of English and Brazilian Portuguese. |
| Outcome: | The proposed corpus improves on the English and Brazilian Portuguese languages. |
Viability of Machine Translation for Healthcare in Low-Resourced Languages (2025.emnlp-main)
Copied to clipboard
Hellina Hailu Nigatu, Nikita Mehandru, Negasi Haile Abadi, Blen Gebremeskel, Ahmed Alaa, Monojit Choudhury
| Challenge: | MT errors are more pronounced in low-resourced languages where human translators are scarce and MT tools perform poorly. |
| Approach: | They propose to use a publicly available machine translation system to analyze machine translation errors in healthcare domains. |
| Outcome: | The proposed system reduces errors in two low-resourced languages for healthcare. |
A Straightforward Approach to Narratologically Grounded Character Identification (2020.coling-main)
Copied to clipboard
| Challenge: | Existing definitions of character are based on simplified or implicit definitions that do not capture essential distinctions between characters and other referents in narratives. |
| Approach: | They propose a narratologically grounded definition of character that is based on clear narrological principles and annotated 170 narrative texts. |
| Outcome: | The proposed definition of character is based on clear narratological principles and can be reliably annotated (0.78 Cohen’s ). |
Embeddings for Named Entity Recognition in Geoscience Portuguese Literature (2020.lrec-1)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a task within the field of Natural Language Processing that deals with the identification and categorization of Named entities (NEs) in a given text. |
| Approach: | They propose to use vector and tensor embeddings to train Portuguese Named Entity Recognition (NER) in the Geology domain. |
| Outcome: | The proposed model achieves state-of-the-art for the Portuguese Geology domain with one of its embeddings. |
AGILe: The First Lemmatizer for Ancient Greek Inscriptions (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing models for ancient Greek inscriptions are not performant on epigraphic data due to language differences . a lemmatizer for ancient inscription data can enable meaningful generalizations, we show . |
| Approach: | They propose to train an automatic lemmatizer for ancient Greek inscriptions with 80% accuracy . they also show that existing models are not performant on epigraphic data . |
| Outcome: | The proposed model achieves above 80% accuracy on epigraphic data, and makes it available to the community. |
Local Interpretation of Transformer Based on Linear Decomposition (2023.acl-long)
Copied to clipboard
| Challenge: | Existing work on local explanation generation attempts to understand model dynamics on word-level or phraselevel by assigning importance scores on input features. |
| Approach: | They propose to interpret neural networks by linear decomposition by a Transformer model on a single input and a linear decomposing of the output to generate local explanations. |
| Outcome: | The proposed method achieves competitive performance in sentiment classification and machine translation, and fidelity of explanation. |
Are Multimodal LLMs Robust Against Adversarial Perturbations? RoMMath: A Systematic Evaluation on Multimodal Math Reasoning (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent-released MLLMs have shown remarkable performance on various multimodal math reasoning benchmarks. |
| Approach: | They introduce RoMMath, the first benchmark designed to evaluate the capabilities and robustness of multimodal large language models in handling multimodal math reasoning. |
| Outcome: | The proposed model performs well on a broad spectrum of 17 MLLMs and demonstrates that they are robust to adversarial perturbations. |
SLURP: A Spoken Language Understanding Resource Package (2020.emnlp-main)
Copied to clipboard
| Challenge: | Publicly available datasets for Spoken Language Understanding (SLU) are limited. |
| Approach: | They propose a publicly available SLU resource package that includes a multi-domain dataset in English spanning 18 domains. |
| Outcome: | The proposed dataset is bigger and more diverse than existing datasets. |
CoNTACT: A Dutch COVID-19 Adapted BERT for Vaccine Hesitancy and Argumentation Detection (2022.coling-1)
Copied to clipboard
| Challenge: | CoNTACT is a Dutch language model adapted to the domain of COVID-19 tweets . a turbulent vaccine debate has emerged between advocates and opponents of vaccines - a polarization that will continue to influence future views on vaccines. |
| Approach: | They propose a Dutch language model adapted to the domain of COVID-19 tweets . they use 2.8M Dutch COVId-19 related tweets posted in 2021 to test the model . |
| Outcome: | The proposed model shows statistically significant gains over RobBERT on two tasks. |
Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving (2025.findings-acl)
Copied to clipboard
| Challenge: | a benchmark for university-level physics problem solving contains 1,297 expert-annotated problems . a proprietary model, o3-mini, achieves only 59.9% accuracy, highlighting fundamental weaknesses in scientific reasoning, conceptual understanding, and mathematical precision. |
| Approach: | They introduce Physics, a benchmark for university-level physics problem solving. |
| Outcome: | The proposed model achieves only 59.9% accuracy on the most advanced model, o3-mini . the proposed model is a powerful tool for evaluating models on advanced problems . |
Extracting Biomedical Entities from Noisy Audio Transcripts (2024.lrec-main)
Copied to clipboard
Nima Ebadi, Kellen Morgan, Adrian Tan, Billy Linares, Sheri Osborn, Emma Majors, Jeremy Davis, Anthony Rios
| Challenge: | Named Entity Recognition (NER) is particularly affected by noise, often termed the ASR-NLP gap. |
| Approach: | They propose a dataset to bridge the ASR-NLP gap in the biomedical domain by extracting adverse drug reactions and mentions of entities from the Brief Test of Adult Cognition by Telephone (BTACT) exam. |
| Outcome: | The proposed method can clean 2,000 clean and noisy recordings and eliminate errors using zero-shot and few-shot methods. |
Improving Crowdsourcing-Based Annotation of Japanese Discourse Relations (L18-1)
Copied to clipboard
| Challenge: | Discourse parsing is an important task in natural language processing, but few languages have corpora annotated with discourse relations . crowdsourcing-based annotations are of poor quality and require expensive and time-consuming . et al. (2009) evaluated the quality of annotations using expert annotations. |
| Approach: | They construct a Japanese corpus with discourse annotations through crowdsourcing . they propose improvement techniques based on language tests . |
| Outcome: | The proposed methods improve the quality of the annotations, and will make them publicly available. |
2kenize: Tying Subword Sequences for Chinese Script Conversion (2020.acl-main)
Copied to clipboard
| Challenge: | Traditional Chinese character conversion is a common step in Chinese NLP but current methods do not take into account that a simplified Chinese character can correspond to multiple traditional characters. |
| Approach: | They propose a model that can disambiguate between mappings and convert between the two scripts by using subword segmentation and two language models. |
| Outcome: | The proposed model outperforms previous Chinese Character conversion approaches by 6 points in accuracy. |
A Two-Step Approach for Implicit Event Argument Detection (2020.acl-main)
Copied to clipboard
| Challenge: | et al., 2015) only consider local arguments in the same sentence of the event trigger. |
| Approach: | They propose to decompose the implicit event argument detection task into two sub-problems . they propose to use argument head-word detection and head-to-span expansion to reduce the number of candidates. |
| Outcome: | The proposed model achieves better performance than a strong sequence labeling baseline. |
Investigating Multilingual Coreference Resolution by Universal Annotations (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing systems for multilingual coreference resolution have been challenging due to linguistic diversity and complexity of different languages. |
| Approach: | They propose a multilingual coreference dataset with universal morphosyntactic and coreference annotations. |
| Outcome: | The proposed dataset improves the baseline system by 0.9% . the proposed dataset is based on the framework of Universal Dependencies 2 . |
The Paradigm Discovery Problem (2020.acl-main)
Copied to clipboard
| Challenge: | a paradigm discovery problem is a task of learning an inflectional morphological system from unannotated sentences. |
| Approach: | They formalize the paradigm discovery problem and develop evaluation metrics for judging systems . they use word embeddings and string similarity to cluster forms by cell and by paradigm . |
| Outcome: | The proposed system suggests clustering by cell across different inflection classes is the most pressing challenge for future work. |
Improving the Detection of Multilingual Online Attacks with Rich Social Media Data from Singapore (2023.acl-long)
Copied to clipboard
Janosch Haber, Bertie Vidgen, Matthew Chapman, Vibhor Agarwal, Roy Ka-Wei Lee, Yong Keong Yap, Paul Röttger
| Challenge: | Toxic content is a global problem, but most resources for detecting toxic content are in English . new datasets and models for non-English languages focus exclusively on one language or dialect . |
| Approach: | They propose to use a multilingual dataset of online attacks to identify code-mixed toxic content in Singapore . they collect reddit comments in Indonesian, Malay, Singlish, and other languages and provide fine-grained hierarchical labels for attacks . |
| Outcome: | The proposed dataset provides fine-grained hierarchical labels for online attacks in Singapore . it shows that the metadata can be used for granular error analysis . |
Evaluating Test-Time Scaling LLMs for Legal Reasoning: OpenAI o1, DeepSeek-R1, and Beyond (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experimental results show that Legal-R1 delivers competitive performance across diverse tasks. |
| Approach: | They propose to evaluate 12 large language models across 17 legal tasks across statutory and case-law traditions to determine their general reasoning performance. |
| Outcome: | The proposed model performs well across 17 legal tasks across statutory and case-law traditions. |
Predicting the Focus of Negation: Model and Error Analysis (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that a scope detector can predict the focus of negation . negation is a complex phenomenon present in all human languages . |
| Approach: | They propose to leverage a scope detector to introduce the scope of negation as an additional input to the neural network. |
| Outcome: | The proposed model obtains the best results to date, and analyzes errors depending on scope and context information. |
A Large-Scale Japanese Dataset for Aspect-based Sentiment Analysis (2022.lrec-1)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) has not been explored in the Japanese language . there is no standard Japanese dataset available for ABSA task in the language - a paper by cnn. |
| Approach: | They propose to use a Japanese aspect-based sentiment analysis dataset for hotel reviews domain . they propose to include 53,192 review sentences with seven aspect categories and two polarity labels . |
| Outcome: | The proposed dataset contains 53,192 review sentences with seven aspect categories and two polarity labels. |
On Classifying whether Two Texts are on the Same Side of an Argument (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to same side stance classification (S3C) require domain knowledge and semantic inference to solve the task. |
| Approach: | They propose to use same side stance classification to predict whether two arguments argue for the same stance for a given pair of arguments. |
| Outcome: | The proposed model fails to generalize both within and across topics and domains when adjusting the sampling strategy to a more adversarial scenario. |
Integrating Headedness Information into an Auto-generated Multilingual CCGbank for Improved Semantic Interpretation (2024.lrec-main)
Copied to clipboard
| Challenge: | Combinatory Categorial Grammar is a grammar formalism that provides a transparent interface between syntax and semantics. |
| Approach: | They propose an algorithm that adds semantic representations to existing CCG derivations by combining them with predefined combinatory rules. |
| Outcome: | The proposed method produces bare CCG derivations without any accompanying semantic representations and limits its general applicability. |
Tox-BART: Leveraging Toxicity Attributes for Explanation Generation of Implicit Hate Speech (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing language models to generate implicit hate explanations are lacking in many fields. |
| Approach: | They propose to use language models to generate explicit hate posts to make it clear . they find that simpler models incorporating external toxicity signals outperform KG-infused models . |
| Outcome: | The proposed setup produces more precise explanations than zero-shot GPT-3.5, highlighting the intricate nature of the task. |
ULTRABENCH: Benchmarking LLMs under Extreme Fine-grained Text Generation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing benchmarks evaluate models on only a few attributes, typically fewer than five . a new benchmark evaluates large language models under dense, multi-attribute constraints . |
| Approach: | They propose a benchmark that evaluates large language models under dense, multi-attribute constraints. |
| Outcome: | The proposed benchmark evaluates large language models under dense, multi-attribute constraints. |
Learning to Rank Salient Content for Query-focused Summarization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Query-focused summarization (QFS) is gaining prominence in research community. |
| Approach: | They propose to integrate Learning-to-Rank (LTR) with Query-focused Summarization (QFS) to enhance the summary relevance via content prioritization. |
| Outcome: | The proposed model outperforms the state-of-the-art on QMSum benchmark and SQuALITY benchmark while offering a lower training overhead. |
SAMULE: Self-Learning Agents Enhanced by Multi-level Reflection (2025.emnlp-main)
Copied to clipboard
| Challenge: | Modern AI agents rely on Large Language Models (LLMs) as their reasoning engines, but they still face the challenge of generating meaningful reflections due to inadequate error analysis and a reliance on rare successful trajectories. |
| Approach: | They propose a framework for self-learning agents powered by a retrospective language model that generates reflections during inference. |
| Outcome: | The proposed framework outperforms reflection-based baselines on three challenging benchmarks. |
From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MarkerGen (2025.acl-long)
Copied to clipboard
Peiwen Yuan, Chuyi Tan, Shaoxiong Feng, Yiwei Li, Xinglin Wang, Yueqi Zhang, Jiayi Shi, Boyuan Pan, Yao Hu, Kan Li
| Challenge: | Existing methods to control text length are lacking in LCTG, posing a major limitation for practical applications. |
| Approach: | They propose a plug-and-play approach that decomposes LCTG sub-abilities with human patterns as reference and performs detailed error analysis. |
| Outcome: | The proposed method significantly improves LCTG across various settings, exhibiting outstanding effectiveness and generalizability. |
On Event Individuation for Document-Level Information Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | a bomb exploded in a restaurant in Lima, and a second device was deactivated by the police . |
| Approach: | They argue that the task demands definitive answers to thorny questions of *event individuation* they argue that even human experts disagree on the task . |
| Outcome: | The proposed task demands definitive answers to thorny questions of *event individuation* . the proposed task also raises concerns about the usefulness of template filling metrics . |
Large Language Models as Financial Data Annotators: A Study on Effectiveness and Efficiency (2024.lrec-main)
Copied to clipboard
Toyin D. Aguda, Suchetha Siddagangappa, Elena Kochkina, Simerjot Kaur, Dongsheng Wang, Charese Smiley
| Challenge: | Large Language Models (LLMs) have demonstrated remarkable performance in data annotation tasks on general domain datasets, but their effectiveness on domain specific datasets remains under-explored. |
| Approach: | They compare the annotations produced by three LLMs against expert annotators and crowdworkers. |
| Outcome: | The proposed models outperform expert crowdworkers and crowd-sourced annotators on domain specific datasets. |
Exploring the Boundaries of GPT-4 in Radiology (2023.emnlp-main)
Copied to clipboard
Qianchu Liu, Stephanie Hyland, Shruthi Bannur, Kenza Bouzid, Daniel Castro, Maria Wetscherek, Robert Tinn, Harshita Sharma, Fernando Pérez-García, Anton Schwaighofer, Pranav Rajpurkar, Sameer Khanna, Hoifung Poon, Naoto Usuyama, Anja Thieme, Aditya Nori, Matthew Lungren, Ozan Oktay, Javier Alvarez-Valle
| Challenge: | Recent success of general-domain large language models has changed the natural language processing paradigm towards a unified foundation model across domains and applications. |
| Approach: | They evaluate the performance of GPT-4 on a variety of radiology tasks . they find it outperforms or matches current SOTA radiology models . |
| Outcome: | The proposed model outperforms or matches current SOTA radiology models on a range of tasks. |
Lemmatisation of Medieval Greek: Against the Limits of Transformer’s Capabilities? (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing lemmatisation algorithms display an accuracy drop of around 30pp when tested on unedited, Byzantine Greek epigrams. |
| Approach: | They propose to use transformer-based embeddings and a dictionary look-up to lemmatise unedited, Byzantine Greek epigrams. |
| Outcome: | The proposed method outperforms existing methods and provides detailed error analysis revealing why unedited, Byzantine Greek is so challenging for lemmatisation. |
Probing LLMs for Multilingual Discourse Generalization Through a Unified Label Set (2025.acl-long)
Copied to clipboard
| Challenge: | Existing work on discourse understanding is constrained by framework-dependent discourse representations. |
| Approach: | They examine whether large language models capture discourse knowledge that generalizes across languages and frameworks. |
| Outcome: | The proposed model can generalize discourse information across languages and frameworks. |
Dialect-to-Standard Normalization: A Large-Scale Multilingual Evaluation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text normalization is a range of tasks that consist in replacing non-standard spellings with their standard equivalents. |
| Approach: | They introduce dialect-to-standard normalization as a sentence-level character transduction task and provide a large-scale analysis of these methods. |
| Outcome: | The proposed model performs best for Finnish, Swiss German and Slovene while the pre-trained model using full sentences performs the best for Norwegian. |
VarBench: Robust Language Model Benchmarking Through Dynamic Variable Perturbation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent benchmarks release only training and validation sets, keeping the test set labels closed-source. |
| Approach: | They propose to extract variables from each test case and define a value range for each variable. |
| Outcome: | The proposed method improves the accuracy of the evaluations on four datasets covering mathematical generation and multiple-choice tasks. |
Can Multimodal Foundation Models Understand Schematic Diagrams? An Empirical Study on Information-Seeking QA over Scientific Papers (2025.findings-acl)
Copied to clipboard
| Challenge: | MISS-QA is the first benchmark specifically designed to evaluate the ability of models to interpret schematic diagrams within scientific literature. |
| Approach: | They propose an automated evaluation protocol powered by open-source LLMs trained on human-scored data to ensure reliable evaluation. |
| Outcome: | The proposed protocol is powered by open-source LLMs trained on human-scored data. |
ZIPA: A family of efficient models for multilingual phone recognition (2025.acl-long)
Copied to clipboard
| Challenge: | IPA transcriptions capture major articulatory contrasts in speech sounds, including the voicing status, place of articulation, manner of voicing, and tongue positions. |
| Approach: | They present ZIPA, a family of efficient speech models that advances the state-of-the-art performance of crosslinguistic phone recognition. |
| Outcome: | The proposed model outperforms existing phone recognition systems on 17,000+ hours of normalized phone transcriptions and a novel evaluation set capturing unseen languages and sociophonetic variation. |
“What do you call a dog that is incontrovertibly true? Dogma”: Testing LLM Generalization through Humor (2025.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown strong performance in NLP tasks like text summarization and question answering. |
| Approach: | They propose a new humor-based question-answering benchmark to assess LLMs’ reasoning through carefully crafted puns. |
| Outcome: | Experiments on pun comprehension, resolution, and generation reveal that most LLMs struggle with generalization, even on simple tasks, consistently underperforming the human baseline. |
Detecting Online Community Practices with Large Language Models: A Case Study of Pro-Ukrainian Publics on Twitter (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for identifying practices within social media are not yet available. |
| Approach: | They propose a methodological workflow for computational identification of such practices within social media texts by using open-source models and OpenAI’s large language models. |
| Outcome: | The proposed method improves accuracy and supports context-sensitive moderation and advancing the understanding of online community dynamics. |
VisFinEval: A Scenario-Driven Chinese Multimodal Benchmark for Holistic Financial Understanding (2025.emnlp-main)
Copied to clipboard
Zhaowei Liu, Xin Guo, Haotian Xia, Lingfeng Zeng, Fangqi Lou, Jinyi Niu, Mengping Li, Qi Qi, Jiahuan Li, Wei Zhang, Yinglong Wang, Weige Cai, Weining Shen, Liwen Zhang
| Challenge: | Existing benchmarks focus on text comprehension, but MLLMs lack the ability to integrate visual data over financial visuals. |
| Approach: | They evaluate 21 state-of-the-art multimodal large language models in a zero-shot setting . they use an annotated question–answer pair from eight common financial image modalities . |
| Outcome: | The new benchmark outperforms existing models but trailed financial experts by 14 percentage points. |
Revealing the impact of synthetic native samples and multi-tasking strategies in Hindi-English code-mixed humour and sarcasm detection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, we tried native sample mixing, multi-task learning, and prompting and instruction finetuning very large multilingual language models (VMLMs). |
| Approach: | They used native sample mixing, multi-task learning and prompting and instruction finetuning to improve code-mixed humour and sarcasm detection. |
| Outcome: | The proposed methods improve humour and sarcasm detection by adding native samples to training sets and multitask learning and prompting and instruction finetuning VMLMs. |
SPLICE: A Singleton-Enhanced PipeLIne for Coreference REsolution (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing attempts to integrate singleton mention detection into end-to-end coreference resolution for English have been hampered by the lack of singletont mention spans in the OntoNotes benchmark. |
| Approach: | They propose a two-step neural mention and coreference resolution system that integrates singleton mentions with OntoNotes syntax trees to achieve a near approximation of the Ontonotes dataset with all singletont mentions. |
| Outcome: | The proposed system achieves 94% recall on a sample of gold singletons. |
MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms (2026.findings-acl)
Copied to clipboard
Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu, Chen Xing
| Challenge: | Existing evaluation benchmarks for LLM unit test generation focus on function-level code rather than on more practical, challenging multi-file codebases. |
| Approach: | They propose a multi-file-level benchmark for unit test generation covering Python, Java, and JavaScript. |
| Outcome: | The proposed benchmarks show that most LLMs exhibit moderate performance on MultiFileTest, highlighting the benchmark’s inherent difficulty. |
R2D2: Remembering, Replaying and Dynamic Decision Making with a Reflective Agentic Memory (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods for web agents struggle with efficient navigation and action execution due to limited visibility and understanding of web structures. |
| Approach: | They propose a framework that integrates memory-enhanced navigation and reflective learning to improve web agents' performance. |
| Outcome: | The proposed framework shows significant improvements over existing methods, including 50% reduction in navigation errors and threefold increase in task completion rates. |
RLSeek: Evidence-Grounded Reasoning for RAG Hallucination Detection (2026.acl-long)
Copied to clipboard
Zhaoheng Huang, Dacheng Wen, Yutao Zhu, Xiaoying Lian, Yushi Liang, Kai Hao, Nan Li, Liangjie Zhang, Qi Zhang, Ji-Rong Wen, Zhicheng Dou, Fangzhao Wu
| Challenge: | Recent work addresses this problem by training span-level hallucination detectors using reinforcement learning and chain-of-thought reasoning. |
| Approach: | They propose a framework that explicitly enforces active evidence seeking during CoT reasoning by requiring quotation of relevant source segments at each verification step. |
| Outcome: | The proposed framework improves hallucination span detection performance with limited reasoning overhead and improved robustness in out-of-domain settings. |
Who Said What: Formalization and Benchmarks for the Task of Quote Attribution (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for quote attribution are poorly understood, despite advances in research . previous approaches have used hand-crafted features to identify speaker names . |
| Approach: | They formalize the task of quote attribution and establish a basis for comparison . they compare CEQA and ChatGPT models on available datasets in both English and Chinese . |
| Outcome: | The proposed model outperforms all supervised methods on English and Chinese datasets. |
TounsiBench: Benchmarking Large Language Models for Tunisian Arabic (2025.emnlp-main)
Copied to clipboard
| Challenge: | a dataset of Tunisian Arabic instructions and prompts is used to evaluate LLMs' ability to understand and generate responses in Tunisia . we assess the quality, correctness, relevance, and dialectal adherence of LLM responses . |
| Approach: | They propose a benchmark for evaluating the capabilities of large language models in Tunisian Arabic . they use a dataset of Tunisia Arabic instructions and prompts to evaluate their models . |
| Outcome: | The proposed model can judge quality, correctness, relevance, and dialectal adherence . the model can also generate a leaderboard for the Tunisian Arabic language . |
NSF-SciFy: Mining the NSF Awards Database for Scientific Claims (2026.acl-long)
Copied to clipboard
| Challenge: | NSF-SciFy contains 2.8 million claims from 400,000 abstracts spanning all science and mathematics disciplines. |
| Approach: | They propose to use a dataset to extract scientific claims from National Science Foundation award abstracts and to use it to refine language models. |
| Outcome: | The proposed method improves non-technical abstract generation, claim extraction, and investigation proposal extraction tasks while maintaining high precision and lower recall. |